{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BootstrapOutOfBag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An implementation of the out-of-bag bootstrap to evaluate supervised learning algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `from mlxtend.evaluate import BootstrapOutOfBag`    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Originally, the bootstrap method aims to determine the statistical properties of an estimator when the underlying distribution was unknown and additional samples are not available. Now, in order to exploit this method for the evaluation of predictive models, such as hypotheses for classification and regression, we may prefer a slightly different approach to bootstrapping using the so-called Out-Of-Bag (OOB) or Leave-One-Out Bootstrap (LOOB) technique. Here, we use out-of-bag samples as test sets for evaluation instead of evaluating the model on the training data. Out-of-bag samples are the unique sets of instances that are not used for model fitting as shown in the figure below [1].\n",
    "\n",
    "![](BootstrapOutOfBag_files/bootstrap_concept.png)\n",
    "\n",
    "\n",
    "The figure above illustrates how three random bootstrap samples drawn from an exemplary ten-sample dataset ($X_1,X_2, ..., X_{10}$) and their out-of-bag sample for testing may look like. In practice, Bradley Efron and Robert Tibshirani recommend drawing 50 to 200 bootstrap samples as being sufficient for reliable estimates [2]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "- [1]  https://sebastianraschka.com/blog/2016/model-evaluation-selection-part2.html\n",
    "- [2] Efron, Bradley, and Robert J. Tibshirani. An introduction to the bootstrap. CRC press, 1994. Management of Data (ACM SIGMOD '97), pages 265-276, 1997."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1 -- Evaluating the predictive performance of a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `BootstrapOutOfBag` class mimics the behavior of scikit-learn's cross-validation classes, e.g., `KFold`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4 2 1 3 3] [0]\n",
      "[2 4 1 2 1] [0 3]\n",
      "[4 3 3 4 1] [0 2]\n"
     ]
    }
   ],
   "source": [
    "from mlxtend.evaluate import BootstrapOutOfBag\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "oob = BootstrapOutOfBag(n_splits=3)\n",
    "for train, test in oob.split(np.array([1, 2, 3, 4, 5])):\n",
    "    print(train, test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consequently, we can use `BootstrapOutOfBag` objects via the `cross_val_score` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.96078431  0.92156863  0.95833333]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "lr = LogisticRegression()\n",
    "\n",
    "print(cross_val_score(lr, X, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.92727273  0.96226415  0.94444444]\n"
     ]
    }
   ],
   "source": [
    "print(cross_val_score(lr, X, y, cv=BootstrapOutOfBag(n_splits=3, random_seed=456)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, it is recommended to run at least 200 iterations, though:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean accuracy: 94.8%\n"
     ]
    }
   ],
   "source": [
    "print('Mean accuracy: %.1f%%' % np.mean(100*cross_val_score(\n",
    "    lr, X, y, cv=BootstrapOutOfBag(n_splits=200, random_seed=456))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the bootstrap, we can use the percentile method to compute the confidence bounds of the performance estimate. We pick our lower and upper confidence bounds as follows:\n",
    "\n",
    "- $ACC_{lower}$ = $\\alpha_1th$ percentile of the $ACC_{boot}$ distribution\n",
    "- $ACC_{lower}$ = $\\alpha_2th$ percentile of the $ACC_{boot}$ distribution\n",
    "\n",
    "where $\\alpha_1 = \\alpha$ and $\\alpha_2 = 1-\\alpha$, and the degree of confidence to compute the $100 \\times (1-2 \\times \\alpha)$ confidence interval. For instance, to compute a 95% confidence interval, we pick $\\alpha=0.025$ to obtain the 2.5th and 97.5th percentiles of the *b* bootstrap samples distribution as the upper and lower confidence bounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAD8CAYAAAC4lecIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF4VJREFUeJzt3X2QVfWd5/H3F+ixXbE0hocSUJukEAPKky2LEiOCG6yk\nEkdNpTCUiiaSlEl2rMrDqBVKqrYwppKZJGs22ZCVgjEmWRM1ouVkRCbGMRlFGFsFUSGxDc2qoFmN\nMZHw8N0/+sI20NC3+/bDr+n3q+pWn/M753fv9/76wKfPueeeE5mJJEkq06C+LkCSJB2aQS1JUsEM\nakmSCmZQS5JUMINakqSCGdSSJBXMoJYkqWAGtSRJBTOoJUkq2JC+LgBg2LBh2dDQ0NdlSJLUa9at\nW/daZg7vaL0igrqhoYG1a9f2dRmSJPWaiHipmvU89C1JUsEMakmSCmZQS5JUsCI+o27Pzp07aWlp\n4Z133unrUtRJ9fX1jBkzhrq6ur4uRZL6vQ6DOiJOAv4JGAkksDQzvx0Ri4FrgO2VVW/MzAcqfW4A\nPgnsBv5rZv5LZwtraWnh2GOPpaGhgYjobHf1kczk9ddfp6WlhbFjx/Z1OZLU71WzR70L+EJm/kdE\nHAusi4hVlWXfzMxvtF05IiYA84CJwCjgoYg4NTN3d6awd955x5DuhyKCd7/73Wzfvr3jlSVJHerw\nM+rMfDkz/6My/RawERh9mC4XAT/JzB2Z+SKwGZjeleIM6f7J35skdZ9OnUwWEQ3AVODxStPnI+Lp\niFgWEe+qtI0GtrTp1sLhg12SJB1C1UEdEUOBu4DrMvOPwPeA9wBTgJeBf+jMC0fEwohYGxFrPUwq\nSVL7qjrrOyLqaA3pOzLzboDMfLXN8h8A91dmtwIntek+ptK2n8xcCiwFaGxszK4UL0mq3X3P93UF\nB/vI+L6uoBwd7lFH6weOtwEbM/Mf27Sf2Ga1i4H1lemVwLyIOCoixgLjgDXdV3LvaW5u5rTTTmPB\nggWceuqpzJ8/n4ceeoiZM2cybtw41qxZw9tvv83VV1/N9OnTmTp1Kvfee+++vueeey7Tpk1j2rRp\n/OY3vwHg4YcfZtasWXzsYx/jtNNOY/78+WT6d4okqX3V7FHPBC4HnomIpkrbjcBlETGF1q9sNQOf\nBsjMDRFxJ/AsrWeMf7azZ3y3Z/ny5SxfvrzWp9lnwYIFLFiwoMP1Nm/ezE9/+lOWLVvGWWedxY9+\n9CMeffRRVq5cyc0338yECROYPXs2y5Yt44033mD69OlccMEFjBgxglWrVlFfX8+mTZu47LLL9l3P\n/Mknn2TDhg2MGjWKmTNn8utf/5r3v//93fbeJElHjg6DOjMfBdo7jfeBw/RZAiypoa6DNDc386tf\n/arbnm/WrFlVrTd27FjOOOMMACZOnMicOXOICM444wyam5tpaWlh5cqVfOMbrd9Se+edd/j973/P\nqFGj+NznPkdTUxODBw/mhRde2Pec06dPZ8yYMQBMmTKF5uZmg1qS1K5ir0x2oIaGBs4777xufb5q\nHHXUUfumBw0atG9+0KBB7Nq1i8GDB3PXXXcxfvz+H6gsXryYkSNH8tRTT7Fnzx7q6+vbfc7Bgwez\na9euGt6JJOlI1m+CutpD1b1t7ty53Hrrrdx6661EBE8++SRTp07lzTffZMyYMQwaNIgVK1awe3fN\nR/8lSQOQN+Wo0aJFi9i5cyeTJk1i4sSJLFq0CIBrr72WFStWMHnyZJ577jmOOeaYPq5UktQfRQln\nHDc2NubeE6322rhxI+973/v6qCLVyt+f1H/49ay+ERHrMrOxo/Xco5YkqWD95jNqSToSlLj3qrK5\nRy1JUsEMakmSCmZQS5JUMINakqSCGdSH8corrzBv3jze+973cuaZZ/KhD32IF154gebmZk4//XQA\n/vrXv3LVVVdxxhlnMHnyZB5++OF9/WfNmsX48eOZMmUKU6ZMYdu2bX30Tqpz88037zd/zjnnAOz3\nfiVJvcugPoTM5OKLL2bWrFn89re/Zd26dXz1q1/l1Vdf3W+9H/zgBwA888wzrFq1ii984Qvs2bNn\n3/I77riDpqYmmpqaGDFiRLfX2Pa1anVgUO+945ckqe8Y1Ifwy1/+krq6Oj7zmc/sa5s8eTLnnnvu\nfus9++yzzJ49G4ARI0Zw/PHHc+DFWw5n8eLFXH755Zx99tmMGzduX/ADfP3rX+ess85i0qRJ3HTT\nTUDr3u348eO54oorOP3009myZQu/+MUvmDZtGpMnT2bOnDkAh7z95vLly7nkkku48MILGTduHF/+\n8pcBuP766/nLX/7ClClTmD9/PgBDhw49qN7du3fzpS99aV9d3//+96t+r5Kkzus336Ou9TaXnb1W\n+Pr16znzzDM7XG/y5MmsXLmSyy67jC1btrBu3Tq2bNnC9OnTAbjyyiupq6vj0ksv5Stf+Qqtt/fe\n39NPP81jjz3G22+/zdSpU/nwhz/M+vXr2bRpE2vWrCEz+ehHP8ojjzzCySefzKZNm1ixYgUzZsxg\n+/btXHPNNTzyyCOMHTuWP/zhDwAsWbKk3dtvAjQ1NfHkk09y1FFHMX78eD7/+c9zyy238J3vfIem\npqaD6mvrtttu47jjjuOJJ55gx44dzJw5kw9+8IOMHTu26rGVJFWv3wR1rbe5rPa2lp119dVXs3Hj\nRhobGznllFM455xzGDx4MNB62Hv06NG89dZbXHrppdx+++1cccUVBz3HRRddxNFHH83RRx/N+eef\nz5o1a3j00Ud58MEHmTp1KgB/+tOf2LRpEyeffDKnnHIKM2bMAOCxxx7jAx/4wL6gPOGEEwB48MEH\n2739JsCcOXM47rjjAJgwYQIvvfQSJ510UlXv98EHH+Tpp5/mZz/7GQBvvvkmmzZtMqglqYf0m6Cu\n9TaX1d7Wcq+JEyfuC6PDGTJkCN/85jf3zZ9zzjmceuqpAIwePRqAY489lk984hOsWbOm3aA+cC87\nIshMbrjhBj796U/vt6y5ubmqG3xkZru333z88cdrus1mZnLrrbcyd+7cqvtIkrqu3wR1b9/mcvbs\n2dx4440sXbqUhQsXAq2HqN9888399j7//Oc/k5kcc8wxrFq1iiFDhjBhwgR27drFG2+8wbBhw9i5\ncyf333//vkPPB7r33nu54YYbePvtt3n44Ye55ZZbOProo1m0aBHz589n6NChbN26lbq6uoP6zpgx\ng2uvvZYXX3xx36HvE0444ZC33zycuro6du7c2e7r7DV37ly+973vMXv2bOrq6njhhRcYPXq0dweT\npB7Sb4K6t0UE99xzD9dddx1f+9rXqK+vp6GhgW9961v7rbdt2zbmzp3LoEGDGD16NLfffjsAO3bs\nYO7cuezcuZPdu3dzwQUXcM0117T7WpMmTeL888/ntddeY9GiRYwaNYpRo0axceNGzj77bKD1xK4f\n/vCH+w6r7zV8+HCWLl3KJZdcwp49exgxYgSrVq1i0aJFXHfddUyaNIk9e/YwduxY7r///sO+54UL\nFzJp0iSmTZvGHXfc0e46n/rUp2hubmbatGlkJsOHD+fnP/95VWMqSeo8b3PZxxYvXszQoUP54he/\n2NeldKuB8vuTOsubclTH21z+f349S5Kkgnnou48tXry4r0uQJBWs6D3qEg7Lq/P8vUlS9yk2qOvr\n63n99df9T7+fyUxef/116uvr+7oUSToiFHvoe8yYMbS0tLB9+/a+LkWdVF9fz5gxY/q6DEk6IhQb\n1HV1dV7tSpI04BV76FuSJBnUkiQVzaCWJKlgBrUkSQUzqCVJKphBLUlSwQxqSZIKZlBLklSwDoM6\nIk6KiF9GxLMRsSEi/q7SfkJErIqITZWf72rT54aI2BwRz0fE3J58A5IkHcmq2aPeBXwhMycAM4DP\nRsQE4HpgdWaOA1ZX5qksmwdMBC4EvhsRg3uieEmSjnQdXkI0M18GXq5MvxURG4HRwEXArMpqK4CH\ngb+vtP8kM3cAL0bEZmA68O/dXbwk6ch03/N9XcH+PjK+7167U59RR0QDMBV4HBhZCXGAV4CRlenR\nwJY23VoqbQc+18KIWBsRa73xhiRJ7as6qCNiKHAXcF1m/rHtsmy9F2Wn7keZmUszszEzG4cPH96Z\nrpIkDRhVBXVE1NEa0ndk5t2V5lcj4sTK8hOBbZX2rcBJbbqPqbRJkqROquas7wBuAzZm5j+2WbQS\nuLIyfSVwb5v2eRFxVESMBcYBa7qvZEmSBo5q7kc9E7gceCYimiptNwK3AHdGxCeBl4CPA2Tmhoi4\nE3iW1jPGP5uZu7u9ckmSBoBqzvp+FIhDLJ5ziD5LgCU11CVJkvDKZJIkFc2gliSpYAa1JEkFM6gl\nSSqYQS1JUsEMakmSCmZQS5JUMINakqSCGdSSJBXMoJYkqWAGtSRJBTOoJUkqmEEtSVLBDGpJkgpm\nUEuSVDCDWpKkghnUkiQVzKCWJKlgBrUkSQUzqCVJKphBLUlSwQxqSZIKZlBLklQwg1qSpIIZ1JIk\nFcygliSpYAa1JEkFM6glSSqYQS1JUsEMakmSCmZQS5JUMINakqSCdRjUEbEsIrZFxPo2bYsjYmtE\nNFUeH2qz7IaI2BwRz0fE3J4qXJKkgaCaPerlwIXttH8zM6dUHg8ARMQEYB4wsdLnuxExuLuKlSRp\noBnS0QqZ+UhENFT5fBcBP8nMHcCLEbEZmA78e5crlKQuuu/5vq5Aql0tn1F/PiKerhwaf1elbTSw\npc06LZU2SZLUBV0N6u8B7wGmAC8D/9DZJ4iIhRGxNiLWbt++vYtlSJJ0ZOtSUGfmq5m5OzP3AD+g\n9fA2wFbgpDarjqm0tfccSzOzMTMbhw8f3pUyJEk64nUpqCPixDazFwN7zwhfCcyLiKMiYiwwDlhT\nW4mSJA1cHZ5MFhE/BmYBwyKiBbgJmBURU4AEmoFPA2Tmhoi4E3gW2AV8NjN390zpkiQd+ao56/uy\ndppvO8z6S4AltRQlSZJaeWUySZIKZlBLklQwg1qSpIIZ1JIkFcygliSpYAa1JEkFM6glSSqYQS1J\nUsEMakmSCmZQS5JUMINakqSCGdSSJBXMoJYkqWAGtSRJBTOoJUkqmEEtSVLBDGpJkgpmUEuSVDCD\nWpKkghnUkiQVzKCWJKlgBrUkSQUzqCVJKphBLUlSwQxqSZIKZlBLklQwg1qSpIIZ1JIkFcygliSp\nYAa1JEkFM6glSSqYQS1JUsEMakmSCtZhUEfEsojYFhHr27SdEBGrImJT5ee72iy7ISI2R8TzETG3\npwqXJGkgqGaPejlw4QFt1wOrM3McsLoyT0RMAOYBEyt9vhsRg7utWkmSBpgOgzozHwH+cEDzRcCK\nyvQK4G/btP8kM3dk5ovAZmB6N9UqSdKA09XPqEdm5suV6VeAkZXp0cCWNuu1VNokSVIXDKn1CTIz\nIyI72y8iFgILAU4++eRay5DUx+57vq8rkI5MXd2jfjUiTgSo/NxWad8KnNRmvTGVtoNk5tLMbMzM\nxuHDh3exDEmSjmxdDeqVwJWV6SuBe9u0z4uIoyJiLDAOWFNbiZIkDVwdHvqOiB8Ds4BhEdEC3ATc\nAtwZEZ8EXgI+DpCZGyLiTuBZYBfw2czc3UO1S5J0xOswqDPzskMsmnOI9ZcAS2opSpIktfLKZJIk\nFcygliSpYAa1JEkFM6glSSqYQS1JUsEMakmSCmZQS5JUMINakqSCGdSSJBXMoJYkqWAGtSRJBTOo\nJUkqmEEtSVLBDGpJkgpmUEuSVDCDWpKkghnUkiQVzKCWJKlgBrUkSQUzqCVJKphBLUlSwQxqSZIK\nZlBLklQwg1qSpIIZ1JIkFcygliSpYAa1JEkFM6glSSqYQS1JUsEMakmSCmZQS5JUMINakqSCGdSS\nJBVsSC2dI6IZeAvYDezKzMaIOAH430AD0Ax8PDP/b21lSpI0MHXHHvX5mTklMxsr89cDqzNzHLC6\nMi9JkrqgJw59XwSsqEyvAP62B15DkqQBodagTuChiFgXEQsrbSMz8+XK9CvAyPY6RsTCiFgbEWu3\nb99eYxmSJB2ZavqMGnh/Zm6NiBHAqoh4ru3CzMyIyPY6ZuZSYClAY2Nju+tIkjTQ1bRHnZlbKz+3\nAfcA04FXI+JEgMrPbbUWKUnSQNXloI6IYyLi2L3TwAeB9cBK4MrKalcC99ZapCRJA1Uth75HAvdE\nxN7n+VFm/iIingDujIhPAi8BH6+9TEmSBqYuB3Vm/g6Y3E7768CcWoqSJEmtvDKZJEkFM6glSSqY\nQS1JUsFq/R61JKkPrL57Oa9ubWbk6AbmXLKgr8tRDzKoJakfWn3PctY/8StOP+s8g/oI56FvSZIK\nZlBLklQwg1qSpIL5GbXUD933fF9XIKm3uEctSVLBDGpJkgpmUEuSVDCDWpKkghnUkiQVzKCWJKlg\nBrUkSQUzqCVJKphBLUlSwQxqSZIKZlBLklQwg1qSpIIZ1JIkFcy7Z0kd8E5VkvqSQS31I6vvXs6r\nW5sZObqBOZcs6OtypAFh9d3LWbejmYaGBhYsWNDrr29QS/3I6nuWs/6JX3H6WecZ1FIv2fvv7rzz\nzuuToPYzakmSCmZQS5JUMINakqSCGdSSJBXMoJYkqWAGtSRJBTOoJUkqWI99jzoiLgS+DQwG/ldm\n3tJTr6Ujh1cBk6T99UhQR8Rg4H8A/wVoAZ6IiJWZ+WxPvJ66xlCUpPL11B71dGBzZv4OICJ+AlwE\n9FpQlxhCHxnf1xVIkvqbnvqMejSwpc18S6VNkiR1QmRm9z9pxMeACzPzU5X5y4H/nJmfa7POQmBh\nZXY8cOA+8DDgtW4vbuByPLufY9q9HM/u5Xh2r54Yz1Myc3hHK/XUoe+twElt5sdU2vbJzKXA0kM9\nQUSszczGnilv4HE8u59j2r0cz+7leHavvhzPnjr0/QQwLiLGRsTfAPOAlT30WpIkHbF6ZI86M3dF\nxOeAf6H161nLMnNDT7yWJElHsh77HnVmPgA8UMNTHPKwuLrE8ex+jmn3cjy7l+PZvfpsPHvkZDJJ\nktQ9vISoJEkF6/WgjogLI+L5iNgcEde3s/xLEdFUeayPiN0RcUJlWXNEPFNZtra3ay9VFWN6XETc\nFxFPRcSGiLiq2r4DUY3j6TZ6gCrG810RcU9EPB0RayLi9Gr7DlQ1jqnbaBsRsSwitkXE+kMsj4j4\n75WxfjoiprVZ1jvbZ2b22oPWE8t+C7wH+BvgKWDCYdb/CPCvbeabgWG9WXPpj2rGFLgR+Fplejjw\nh8q6nfp9DIRHLeNZmXcb7fx4fh24qTJ9GrC62r4D8VHLmFbm3Ub3H6sPANOA9YdY/iHgn4EAZgCP\nV/t76K5Hb+9R77u0aGb+Fdh7adFDuQz4ca9U1n9VM6YJHBsRAQylNVh2Vdl3oKllPHWwasZzAvCv\nAJn5HNAQESOr7DsQ1TKmOkBmPkLrv+FDuQj4p2z1GHB8RJxIL26fvR3UVV9aNCL+E3AhcFeb5gQe\nioh1lSubqbox/Q7wPuD/AM8Af5eZe6rsO9DUMp7gNnqgasbzKeASgIiYDpxC60WS3D7bV8uYgtto\nZx1qvHtt++yxr2d1g48Av87Mtn/pvD8zt0bECGBVRDxX+WtIhzcXaAJmA++ldez+rW9L6tfaHc/M\n/CNuo11xC/DtiGii9Q+fJ4HdfVtSv3e4MXUb7Wd6e4+6w0uLtjGPAw57Z+bWys9twD20HnoY6KoZ\n06uAuyuHbjYDL9L6uVVnfh8DRS3j6TZ6sGouJ/zHzLwqM6cAV9D6uf/vquk7QNUypm6jnXeo8e61\n7bO3g7qqS4tGxHHAecC9bdqOiYhj904DHwTaPUtvgKlmTH8PzAGofE41ntZ/tF7q9WBdHk+30XZ1\nOJ4RcXxlGcCngEcqRyfcPtvX5TF1G+2SlcAVlbO/ZwBvZubL9OL22auHvvMQlxaNiM9Ulv/PyqoX\nAw9m5tttuo8E7mk9f4chwI8y8xe9V32ZqhzT/wYsj4hnaD1z8e8z8zWA9vr2xfsoRS3jGRHvwW10\nP1WO5/uAFRGRwAbgk4fr2xfvoyS1jCn+P3qQiPgxMAsYFhEtwE1AHewbywdoPfN7M/BnWo+o9er2\n6ZXJJEkqmFcmkySpYAa1JEkFM6glSSqYQS1JUsEMakmSCmZQS5JUMINakqSCGdSSJBXs/wGornGj\n1K0dHAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ba42e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracies = cross_val_score(lr, X, y, cv=BootstrapOutOfBag(n_splits=1000, random_seed=456))\n",
    "mean = np.mean(accuracies)\n",
    "\n",
    "lower = np.percentile(accuracies, 2.5)\n",
    "upper = np.percentile(accuracies, 97.5)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8, 4))\n",
    "ax.vlines(mean, [0], 40, lw=2.5, linestyle='-', label='mean')\n",
    "ax.vlines(lower, [0], 15, lw=2.5, linestyle='-.', label='CI95 percentile')\n",
    "ax.vlines(upper, [0], 15, lw=2.5, linestyle='-.')\n",
    "\n",
    "ax.hist(accuracies, bins=11,\n",
    "        color='#0080ff', edgecolor=\"none\", \n",
    "        alpha=0.3)\n",
    "plt.legend(loc='upper left')\n",
    "\n",
    "\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## BootstrapOutOfBag\n",
      "\n",
      "*BootstrapOutOfBag(n_splits=200, random_seed=None)*\n",
      "\n",
      "**Parameters**\n",
      "\n",
      "\n",
      "- `n_splits` : int (default=200)\n",
      "\n",
      "    Number of bootstrap iterations.\n",
      "    Must be larger than 1.\n",
      "\n",
      "\n",
      "- `random_seed` : int (default=None)\n",
      "\n",
      "    If int, random_seed is the seed used by\n",
      "    the random number generator.\n",
      "\n",
      "\n",
      "**Returns**\n",
      "\n",
      "- `train_idx` : ndarray\n",
      "\n",
      "    The training set indices for that split.\n",
      "\n",
      "\n",
      "- `test_idx` : ndarray\n",
      "\n",
      "    The testing set indices for that split.\n",
      "\n",
      "**Examples**\n",
      "\n",
      "For usage examples, please see\n",
      "    [http://rasbt.github.io/mlxtend/user_guide/evaluate/BootstrapOutOfBag/](http://rasbt.github.io/mlxtend/user_guide/evaluate/BootstrapOutOfBag/)\n",
      "\n",
      "### Methods\n",
      "\n",
      "<hr>\n",
      "\n",
      "*get_n_splits(X=None, y=None, groups=None)*\n",
      "\n",
      "Returns the number of splitting iterations in the cross-validator\n",
      "\n",
      "**Parameters**\n",
      "\n",
      "- `X` : object\n",
      "\n",
      "    Always ignored, exists for compatibility with scikit-learn.\n",
      "\n",
      "\n",
      "- `y` : object\n",
      "\n",
      "    Always ignored, exists for compatibility with scikit-learn.\n",
      "\n",
      "\n",
      "- `groups` : object\n",
      "\n",
      "    Always ignored, exists for compatibility with scikit-learn.\n",
      "\n",
      "**Returns**\n",
      "\n",
      "\n",
      "- `n_splits` : int\n",
      "\n",
      "    Returns the number of splitting iterations in the cross-validator.\n",
      "\n",
      "<hr>\n",
      "\n",
      "*split(X, y=None, groups=None)*\n",
      "\n",
      "y : array-like or None (default: None)\n",
      "Argument is not used and only included as parameter\n",
      "for compatibility, similar to `KFold` in scikit-learn.\n",
      "\n",
      "\n",
      "- `groups` : array-like or None (default: None)\n",
      "\n",
      "    Argument is not used and only included as parameter\n",
      "    for compatibility, similar to `KFold` in scikit-learn.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('../../api_modules/mlxtend.evaluate/BootstrapOutOfBag.md', 'r') as f:\n",
    "    s = f.read() \n",
    "print(s)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
